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RECORDING METHOD AND APPARATUS 

Field of the Invention 

The present invention relates to a pen for record- 
ing a selected sequence of characters according to the 
preamble of claim 1, as well as a method for effecting 
5 such a recording according to the preamble of claim 7, 
Background of the Invention 

A reader who reads a text often wishes to marlc the 
most important sections of the text in order to be able 
to easily find them when next reading through the text, 

10 thereby avoiding having to read irrelevant parts of the 
text . It is known to use so-called highlighter pens for 
this purpose . 

A reader may also need to put together selected 
parts of a text into a new document, which can be edit- 

15 ed in a word-processing program in a computer. A known 

method for feeding text into a computer is to use a scan- 
ner. There are stationary scanners and portable scanners. 
A stationary scanner is used to scan whole pages of text, 
the scanner being passed over the text automatically at a 

2C constant speed. This type cf scanner is nci suitable fcr 
scanning selected parts c f a zexz c n a page. A p c r r a b 1 e 
ssanner, on the other hand, can be usee fcr this purpose. 

US 5,301,243 discloses a handheld scanner fc^r read- 
ing characters from a character string on a substrate. 

25 The scanner, which is passed over the characters which 
are to be read in contact with the substrate, nas an 
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is reccrded. The "slices" are stored in the scanner as a 
digital bitmap. Subsequently, OCR software (OCR = Optical 
Character Recognition) is used to identify the scanned 
characters and to store them in character-coded format, 
5 e.g. with the aid of ASCII code, in a memory. The charac- 
ter recognition can be performed either in the scanner or 
in an external computer to which the bitmapped characters 
have been transferred. 

A problem associated with the use of portable scan- 
10 ners is that the scanner has to know rhe distance berween 
the scanned "slices" in order to be able to identify the 
characters correctly. This means that the scanner either 
has to be passed over the characters at a constant speed, 
which is very difficult to achieve manually, or else con- 
15 tain a very accurate distance meter. 

In the above-mentioned US 5,301,243, the distance 
problem is obviated with the aid of a mechanical distance 
meter in the form of a wheel. However, a mechanical dis- 
tance meter comprises moving parts, which is a drawback 
20 as far as durability and manufacturing are concerned. 

Moreover, the distance meter makes it difficult to record 
information from a newspaper or the like if the latter is 
ncr clawed cr. a hard surface. In addition, the distance 
t^Leter impairs the possiitility if seeing the mfcrxaticn 
2 5 which is to be recorded, since the wheel has tc abut 

against the surface from which the information is tc be 
recorded and consequently makes the end of the scanner 
wnich abuts against the surface bulkier. 
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rotated around its longitudinal axis while a sequence of 
characters is being recorded, the scanner may have diffi- 
culty identifying the characters recorded, since there is 
no directional information in the scanned "slices". This 
5 is a serious drawback of the prior art since different 
individuals often hold a scanner in different ways. In 
particular, there is a difference in the way right-handed 
and left-handed individuals hold scanners. Furthermore, 
an individual often changes the position of his hands as 

10 the scanner moves along a line. This is a particular 

problem when one is holding a book or the like in one' s 
hand and wishes to record text. 

US 4,949,391 discloses an optical scanner with no 
distance meter. More specifically, this oprical scanner 

15 comprises a two-dimensional sensor , which records images 
with partially overlapping contents of a text over which 
the scanner is passed. However, the handheld scanner is 
really only an "unintelligent" recording unit which 
records characters without interpreting them. Instead, 

2 0 the OCR processing is carried out in a computer to which 
the scanner is connected . The scanner correlates with 
each other images recorded one after another in order to 
remove columns wirh redundant rnfcrmaricn from the tmaoe::" 
so that they wrll require less storage space. Tne scanner 

2 5 can only be micved m a predetermined direction which is 
defined by a wheel. 
Summary of the Invention 

One object of the invention is to obviate, or at 



selectee sequence z z 3 n a r c i e r '.*/ 1 1 n c ' t t n e :i c z i u i 
.rq p o TTi e t e r and with n c r o u i r e m. e n t c t 5 constant s "c e e d 
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In this connection, a further object is to provide 
such a method and such a pen which can be held rotated 
in different ways around its longitudinal axis and which 
specifically allows the pen to be rotated around its lon- 
5 gitudinal axis while a selected sequence of characters is 
being recorded, without the reliability of the identifi- 
cation of recorded characters being reduced in a decisive 
manner . 

The objects are achieved by a pen and a method for 
10 recording a selected characrer sequence which have the 
features stated in claims 1 and 7, respectively. 

As mentioned above, the pen and the method accord- 
ing to the invention are used for recording a selected 
sequence of characters by passing the pen over the char- 
15 acter sequence. The sequence selected can, for example, 
consist of one or several words or one or several sen- 
tences selected from a longer text. The characters can 
be letters, numbers and or other written characters or 
symjools . The characters can be scanned from a paper or 
20 another medium., which can be scanned optically. 

The pen comprises a light-sensitive sensor with a 
two-dimensional sensor surface. In this context, a two- 
iimensicnal sense r surface r'efers "Co a sensor surface 
which can reccra more rhan one pixel m twc direc::ions 
25 perpendicular tc one another. The sensor is preferably 

of the CCD type. It can record two-dimensional images m 
grey scale or m colour. The images consist of a plura- 
lity of pixels, each being defined by an intensity value, 
-^•y-o ^. im.aaes should be such that there is 



desirec ana available sensors 
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unit, a memory, input/output circuits, and program in- 
structions for implementation of the method according zo 
the invention. The signal processing unit preferably com- 
prises a suitably programmed microprocessor. The signal 
5 processing unit employs the partially overlapping con- 
tents of the images to put the recorded images together 
into one image. The putting-together is done both verti- 
cally and horizontally so that the vertical as well as 
the horizontal directional information in the images is 

10 utilised. As a result, there is no duolication of the 
contents in the composite image. 

When an image has been put together, the signal 
processing unit identifies the characters m the image 
and stores them in character-coded format m a memory in 

15 the pen. This means that the pen is of the stand-alone 

type. However, it is suitably connectable to an external 
computer, so that the information can be transferred to 
the latter. A composite image preferably has a predeter- 
mined maximum length. In order to increase the speed and 

20 reduce the memory requirement for images which have been 
recorded but not yet put together, the putting- together 
is suitably carried out sequentially as the images are 
being recorded. Thus, tne fac:: that the purr ing- tcge tne r 
can be performed m rea^ time reauces the memory require- 

25 m.ent for re::crded images. The putting- logetner normally 
continues until all recorded images have been put toge- 
ther. However, it is interrupted if the predetermined 
maximum length of the composite image is reached. Alter- 
natively, the putting-together can be carried out after 



an indicating device for maicatmc tnc: it i^ reacy tc 
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Bv using a sensor with a two-dimensional sensor sur- 
face; by means of which images with partially overlapping 
contents can be recorded, and by putting together the 
images into one image, it is possible to eliminate the 

5 requirement of a constant movement speed or of the provi- 
sion of a distance meter. By virtue of the contents of 
the images partially overlapping, the images will them- 
selves contain information about their relative position, 
i,e, distance information, and as a result there is no 

10 need to record this informarion using d ux^Ldncje meter. 
Moreover, the user can hold the pen at an optional angle 
of rotation in relation to the text and can even rotate 
the pen while recording the images without impairing the 
character identification, since the overlapping images 

15 contain directional information, which indicates their 
relative position. This considerably improves user- 
friendliness . 

Preferably, the pen has a display, so that the user 
can read the recorded character sequence without first 
20 having to transfer it to an ordinary computer. 

As mentioned above, in order for it to be possible 
to carry out the putting-together, there has to be a cer- 
tain overlapping of the contents of the recorded images. 
The degree of overlapping shoulc preferably be greater 
25 than 20%, preferably greater than 50%. 

To make the contrast in the recorded images suffi- 
ciently great and to avoid blurriness caused by the pen 
being moved in relation to the character sequence, the 
_ ^ ^ V- cr w h :. ^ h. a r be e c o r d e d are o r e f e r a b 1 v i 1 1 u - 



which IS fitted m or to tne apparatus. The frequency of 
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frequency for reading the contents of the light-sensitive 
sensor. A fast mechanical shutter, for example, could be 
used instead of the pulsed illumination. However, this 
results in a more complicated apparatus. 
5 When the images are being put together, images 

recorded one after another are preferably examined in 
pairs with respect to possible overlap positions for 
determining the overlap position which affords the best 
correspondence between the contents of each pair. The 
10 overlap posLtron determined xn thxs misnner rs used when 
putring together the images of the pair. 

When the best overlap position has been determined, 
a pixel intensity value is preferably determined for each 
overlapping pixel. This pixel intensity value is based 
15 on the intensity values of the pixel in question in the 

images of the pair. The pixel intensity values determined 
in this manner are used in the final composite image. 

When the whole image has been put together, the 
image is preferably divided into a plurality of sub- 
20 images each comprising a character. This division, which 
thus does not have to be a physical division but rather 
can be performed in the whole image, constitutes a prepa- 
ration for the actual characrer recognition, each charac- 
ter preferably being interpreted separately. Alternative- 
25 ly, whole words or parts thereof could be interpreted. 

The division of the image into sub-images can be 
effected by the intensity of the pixels being added up 
row by row and column by column in the composite image in 
^rder to obtain a plurality of row sums and a plurality 

site image is advantageously effected with the aid of a 
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Suiuably, in this connection, the intensity value of 
each pixel in a sub-image is determined and is fed as an 
input signal to the neural network, which is adapted to 
identify the character in said sub-image with the aid of 
5 the input signals. 

Prior to the identification of the characters, the 
composite image can be rotated if necessary so that the 
characters in the image form a horizontal row. This faci- 
litates the identification. 
10 The pen according to the invention can for example 

be employed to record text while studying, to scan nume- 
rical values, to compose new documents on the basis of 
old documents which exist in paper form only, and for 
many other applications . 
15 Brief Description of the Drawings 

The present invention will be described below by way 
of an example which illustrates how the invention can be 
realised and used. The description refers to the accom- 
panying drawings, in which 
20 Fig. 1 schematically shows an embodiment of a high- 

lighter pen according to the invention; 

Fig. 2 is a block diagram of the electronic cir- 
cuitry m an embodiment of the highlighter pen acccrdmg 
1 c the invention; 
25 Fig. 3 13 a flowchart and shows an implementation of 

the method according to the invention; 



Figs 4a-4e illustrate an example of an implementa- 
tion of the method according to the invention; 




escrioticn of a Preferred Em.bodiment 



wo 98/20446 




PCT/SE97/01827 



ly the same shape as a conventional highlighter pen. One 
short end of the casing 1 has a window 2, which is in- 
tended to abut against or be held a short distance from 
a storage medium, usually a sheet of paper, from which 
5 characters are to be recorded. The width of the window 2 
is chosen on the basis of the desired image-recording 
frequency, the desired maximum traction speed and avail- 
able sensors. The height of the window 2 is somewhat 
greater that the highest character which is to be record- 
iO ed. The window 2 form^i da dcute angle with the longlLudi- 
nal axis of the pen, so that the user is guided to hold 
the pen at a predetermined angle to the sheet of paper. 
Moreover, the window is somewhat retracted into the cas- 
ing so that it will not wear against the paper. 
15 The casing 1 essentially contains an optical part 3, 

an electronic part 4 and a power supply 5. 

The optical part 3 comprises an LED 6, a lens system 
7 and a light-sensitive sensor 8 which constitutes an 
interface to the electronic part 4. 
20 The LED 6 has the task of illuminating the part of 

the storage medium which is currently under the window. 
A diffuser 9 is mounted in front of the LED 6 to diffuse 
^he light. 

The task of the lens system 7 is to project an image 
25 of the part of the storage medium which is located under 
the window 2 of the light-sensitive sensor 8 as accurate- 
ly as possible. If the sensor 8 has a smaller light-sen- 
sitive area than the window 2, the lens system 7 must 
-.ic^n >-oHi]r-^ -he imaae. The skilled oerson will be able to 

sists of a rwo-dimensional square CCD unit {CCD - charge 
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mounted at a small angle to the window 2 and on its own 
printed circuit board 11. 

The power supply to the pen is obtained from a bar- 
tery 12 which is mounted in a separate compartment 13 in 
5 the casing . 

The block diagram in Fig. 2 schematically shows the 
electronic part 4. It comprises a processor 20, which by 
the intermediary of a bus 21 is connected to a ROM 22, in 
which the programs of the processor are stored, to a 
read/v;rite m.emory which constitutes the working memo- 

ry of the processor and in which the images from the sen- 
sor as well as the identified and interpreted characters 
are stored, to a control logic unit 24, as well as to the 
sensor 8 and to the LED 6. The control logic unit 24 in 
15 turn is connected to a number of peripheral units, which 
comprise a display 2, which is mounted in the casing, an 
IR transceiver 26 for transferring information to/from an 
external computer, buttons 27, by means of which the user 
can control the apparatus, a tracer LED 28 which emits a 
20 light beam which makes it easier for the user to follow 

the text, and an indicating device 29, e.g. an LED, indi- 
cating that the pen is ready to record characters. Con- 
trol signals tc the memories, the sensor, and the peri- 
cheral units are generated in the ccntrol logic unir 24. 
25 The control logic unit also handles generation and pric- 
ritisation of interruptions to the processor. The buttons 
27, the IR transceiver 26, the display 25 and the tracer 
LED 28, and the LED 6 are accessed by the processor writ- 
ing and reading in the register in the control logic 



aaamst a sneet of paper with text m tne location wnere 
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latter over the text he wishes to record, following the 
text in the same way as when one reads a text. The tracer 
LED 28 emits a light beam which facilitates following the 
lines. When the user activates the pen, the processor 20 
5 commands the LED 6 to begin generating strobe pulses at a 
predetermined frequency of 25 Hz, the images recorded by 
the sensor being stored in the read/write memory 23 at 
the same frequency. However, text which is stored in the 
form of images requires a large amounr of memory space 

10 and cannot immediately be used by ^ r.nmpnter in, for 
example, a word-processing program. Consequently, the 
characters in the images are identified and stored with 
the aid of ASCII code. When the user has passed the pen 
over a selected sequence of characters or has come to 

15 the end of a line , he lifts the pen from the paper and 
releases the activating button, whereby the processor 
20 turns off the LED 6. When the selected character 
sequences have been recorded, the user can control the 
pen to show recorded text on the display 2 5 or to trans- 

20 fer the text to an external computer by the intermediary 
of the IR transceiver 12. The possibility of showing the 
scanned character information directly on the display has 
o r o V G n very essential since a user often v; i s h e s to verify 
that the right character has been scanned. 

25 The flowchart m Fig. 3 shows an implementation of 

the method according to the invention. In step 301, 
images with overlapping contents are scanned with the 
aid of the pen described above and are storea in a cur- 
rent image area m the read/write memory 23. The images 
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an image is available. If there is no previous image, the 
current image is entered directly into a line image area 
in the read/write memory. 

In order to determine how the current image is to 
5 be put together with the previous image so that the best 
correspondence is achieved between the contents of the 
images, every possible overlap position between the 
images is examined, at the pixel level, and an overlap 
measurement is determined as follows: 
]_ Q j_ ) Fui- each overlapping pixel posrtu.cn, tne grey 

scale values of the two pixels included are added up if 
the latter are not white. A pixel position in which none 
of the pixels are white is designated a plus position. 

2) The grey scale sums for all the plus positions 

15 are added up. 

3) The neighbours of each pixel position are exa- 
mined. If an overlapping pixel position is not a neigh- 
bour of a plus position and consists of a pixel which is 
white and a pixel position which is not white, the grey 

20 scale value of the non-white pixel is subtracted, pos- 
sibly multiplied by a constant, from the sum in point 2) . 

4) The overlap position which provides the highest 
overlap measurement as stared above rs selecrea. In the 
resulrina ccmoosite image the mean value of tne grey 

25 scale value of the overlapping pixels is used. In this 
way, noise can be suppressed in the overlap area. The 
putting-together is thus carried out both vertically and 
horizontally. If it is detected that, when being put 
^oaether, the images do not end up on a horizontal line, 

The ccmpcsire image gradually develops m the line 
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the line image area be big enough to store an A4 line of 
normal typewritten text. 

In step 303, the software of the processor 20 
divides the composite image rn the line image memory area 
5 into sub-images each containing only one character. The 
purpose of this is to create input signals to a neural 
network: software which is to interpret the characters. 
The division is effected by adding up the grey scale 
values of the pixels for each pixel row and each pixel 

10 column in Lhe cuiupu^iLe image. By sLudyiiig the local 
intensity minima for the row sums and column sums ob- 
tained, boundaries for the extent of each character in 
the image can be determined. 

If the composite image contains more than one line 

15 of text, which may be the case if the text is small, the 
lines of text must be separated and the line in question 
must be selected before the division into sub-images is 
carried out. The separation can be performed using a 
technique like the one used for the division into sub- 

2 0 images . 

In step 304, each of the sub-images is scaled down 
to a predetermined pixel format by the sub-image being 

divided inic groups of pixels, each oemg replaced oy 
a pixel whose grey scale value ccrrespc'nds ic tne mean 
25 value of the arey scale values of the p;ixels induced in 
rhe group. If required, a like downscalmg can be carried 
out between other steps in the method according to the 
invention. Further, the character is centred with respect 
i ^ s no'.n^. of balance and the arev scale values are 

ccmiocsite image of the character sequer.ce aepictec is 
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character are fed as input signals to a neural network. 
Each output from the neural network represents a charac- 
ter which the network can identify. The output from the 
network which has the highest output signal is chosen and 
5 the character thus chosen is stored in step 306 using a 
predetermined character code format, for example ASCII 
code, in the read/write memory 23 in a memory area for 
interpreted characters. When the character identification 
and storing in character-coded format is completed, the 
10 processor dcLivates the indicating device 2 9 Lu inform 
the user that it is ready to record a new character 
sequence, step 307, Subsequently, it goes back to step 
301. 

The steps described above are thus carried out by 
15 the processor 20 with the aid of the associated units 
and suitable software. Such software can be created by 
the skilled person with the aid of the above instruc- 
tions . The character recognition is performed with the 
aid of neural network software, which is trained in a 
20 suitable manner. Neural network software is commercially 
available from, for example, MATLAB Neural Network 
Toolbox, The MathWorks Inc., 24 Prime Park Way, Natick, 
MA C1760, USA. 

Figs 4a-4e illustrate hew the methoc according to 
25 the invention works when tne character sequence "Flygande 
backasiner" is scanned. Fig. 4a shows the text on a sheet 
of paper, which also has a certain amount of "dot noise" 
in the form of small black spots. Fig. 4b shows the 
imaaes which are beina recorded with the aid of the sen- 



cf overlapping depends on the traction speed, i.e. the 
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of the sensor are read out. Fig. 4c shows what the whole 
composite image looks like. Note that the image is still 
stored xn the form of pixels. Fig. 4d illustrates rhe 
division of the whole image into sub-images. Fig. 4e 
5 shows the scaled and normalised letters which are used as 
input signals to the neural network. When the method has 
been implemented the text "Flygande backasiner" is stored 
in the read/write memory of the pen as ACSII code. 

In the example shown, the pen was held straight and 
10 was not rotated at all while the character seguencc was 
being recorded. If the pen had been held rotated at a 
constant angle around its longitudinal axis, the images 
would instead have looked like the ones in Fig. 5a. In 
this case, when the images are put together a slanting 
15 image is obtained, see Fig, 5b, which has to be rotated 
in order for the character sequence to be horizontal , 
Fig. 5g. In order to determine how great a rotation is 
to be performed, the position of the characters in the 
composite image is detected in a couple of places. 
2 0 If the pen is rotated around its longitudinal axis 

at the time of recording, the inclination of the images 
will be different. An example of this is shown in 
Fig . ca . When tne images are put together, see Fig. 6b, 
a wave -shaped character sequence is obtamea. It can be 
25 straightened out to a horizontal line by tne position of 
the characters in the compos ite image being detected m a 
plurality of locations and by different segments being 
rotated to different extents {see Fig. 6c) . 
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CLAIMS 

1. A pen for recording a selected sequence of char- 
5 acters by passing the pen over the sequence of charac- 
ters, comprising a light-sensitive sensor (8) for imag- 
ing the characters and a signal processing unit (20-24) 
for identifying the imaged characters and storing them 
in character coded format in the pen, character- 

dimensional sensor surface, the sensor being adapted to 
image the selected character sequence with the aid of a 
plurality of images with partially overlapping contents, 
and that the signal processing unit (20-24) is adapted to 

15 put together said plurality of images both vertically and 
horizontally into a composite image, whereby compensation 
is obtained for different speeds of movement and diffe- 
rent angles of rotation of the pen, and that the identi- 
fication of the imaged characters is performed on the 

20 basis of the composite image. 

2. A pen according to claim 1, further comprising 
illumination means (6), which are adapted to emit light 
pulses for illummatmc characrers which are to be 
recorded . 

25 3. A pen according tc any one of rhe preceding 

claims, wherein the signal processing unit (20-24) com- 
prises a neural network for identification of the char- 
acters . 

4, A oen acccrdmc to any one of the preceding 



5. A pen according to any one of rne preceding 
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indicating that the pen is ready zo scan the character 
sequence . 

6. A pen according to any one of the preceding 
claimS; further comprising a display (25) for shewing 

5 the identified characters. 

7 . A method for recording a selected sequence of 
characters with the aid of a pen, comprising the steps of 

passing the pen over the selected character sequence 
and simultaneously imaging the selected character se- 
10 quence with the aid of a plurality of images, whose con- 
tents partially overlap , 

putting together said plurality of images both ver- 
tically and horizontally into a composite image with the 
aid of a signal processing unit in the pen, whereby com- 
15 pensation is obtained for different speeds of movement 
and different angles of rotation of the pen, 

identifying the characters in the composite image, 

and 

storing the identified characters in the pen with 
20 the aid of a predetermined character code format. 

8. A method according to claim 7, further comprising 
the step of illuminating the characters which are to be 

recorded w i n 1 1 g h r pulses, when t n e ^en is c e i n c p a s s e 
QT/i^ ^ ^ hem . 

25 9. A method according to claim ^ or claim S, wherein 

the step of putting together said plurality of images 
comprises the examination in pairs of images recorded one 
after another wirh respect to possible overlap positiions 
^ ^ ^ ^ ^ ^ T^^^ ^ ^ ^ ^ 1^ "t". h e r 1 a c n o s i r i.. o n w h i. c h afford? ^ h ^ 

ohe images of the pair. 
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18 



15 



10. A method according to any one of claims 7-9, 
^l^erem the putting-together of the recorded images is 
carried out in real time. 

11. A method according to claim 9, wherein the 
recorded images consist of a plurality of pixels each 
being defined by an intensity value, further comprising 
the steps of 

when putting together the images of the pair after 
determining the best overlap position, determining a 
pixel intensity value, preferably a mean value,, for each 
overlapping pixel, which pixel intensity value is based 
on the intensity values of the pixel in question in the 
images of the pair, and 

using the pixel intensity values determined in this 
manner in the composite image. 

12. A method according to any one of claims 7-11, 
wherein the step of identifying the characters comprises 
dividing the composite image into a plurality of sub- 
images each comprising only one character. 

13. A method according to claim 12, wherein the com- 
posite image consists of a plurality of piLxels, each 
being defined by an intensity value, and wherein the step 
of dividina tne ccmtposite image xntc a plurality cf sub- 



row oy row ana column by column m the composite image 
for obtaining a plurality of row sums and col^omn sums as 
well as identifying local minima among the row sums and 
column scorns obtained for dividing the composite image 
into sub-images. 



images comprises 



adding up the intensity cf the pixels 



of identifying tne characters m tne recorded images com- 
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network, which is adapted to identify the character in 
said sub-image with the aid of the input signals. 

15. A method according to any one of claims 7-14, 
further comprising the step of, prior to the identifica 
5 tion of the characters, rotating the composite image so 
that the characters in said image form a horizontal row 
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